{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enviroment: \n",
    "Open AI gym [CartPole v0](https://github.com/openai/gym/wiki/CartPole-v0)\n",
    "\n",
    "\n",
    "\n",
    "### Observation\n",
    "\n",
    "Type: Box(4)\n",
    "\n",
    "| Num  | Observation          | Min      | Max     |\n",
    "| ---- | -------------------- | -------- | ------- |\n",
    "| 0    | Cart Position        | -2.4     | 2.4     |\n",
    "| 1    | Cart Velocity        | -Inf     | Inf     |\n",
    "| 2    | Pole Angle           | ~ -41.8° | ~ 41.8° |\n",
    "| 3    | Pole Velocity At Tip | -Inf     | Inf     |\n",
    "\n",
    "### Actions\n",
    "\n",
    "Type: Discrete(2)\n",
    "\n",
    "| Num  | Action                 |\n",
    "| ---- | ---------------------- |\n",
    "| 0    | Push cart to the left  |\n",
    "| 1    | Push cart to the right |\n",
    "\n",
    "Note: The amount the velocity is reduced or increased is not fixed as it depends on the angle the pole is pointing. This is because the center of gravity of the pole increases the amount of energy needed to move the cart underneath it\n",
    "\n",
    "### Reward\n",
    "\n",
    "Reward is 1 for every step taken, including the termination step\n",
    "\n",
    "### Starting State\n",
    "\n",
    "All observations are assigned a uniform random value between ±0.05\n",
    "\n",
    "### Episode Termination\n",
    "\n",
    "1. Pole Angle is more than ±12°\n",
    "2. Cart Position is more than ±2.4 (center of the cart reaches the edge of the display)\n",
    "3. Episode length is greater than 200\n",
    "\n",
    "### Solved Requirements\n",
    "\n",
    "Considered solved when the average reward is greater than or equal to 195.0 over 100 consecutive trials\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. gym enviroment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.04963332,  0.01977069, -0.03830849,  0.0277437 ])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "env = gym.make(\"CartPole-v0\")\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Q Table setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "LEARNING_RATE = 0.5\n",
    "DISCOUNT = 0.95\n",
    "EPISODES = 50000\n",
    "SHOW_EVERY = 1000\n",
    "Q_TABLE_LEN = 150"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "  return 1 / (1 + np.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/hongtao/anaconda3/envs/spinningup/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: overflow encountered in exp\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "DISCRETE_OS_SIZE = [Q_TABLE_LEN] * (len(env.observation_space.high))\n",
    "\n",
    "\n",
    "observation_high = np.array([env.observation_space.high[0],\n",
    "                    Q_TABLE_LEN*sigmoid(env.observation_space.high[1]),\n",
    "                    env.observation_space.high[2],\n",
    "                    Q_TABLE_LEN*sigmoid(env.observation_space.high[3])])\n",
    "\n",
    "observation_low = np.array([env.observation_space.low[0],\n",
    "                    Q_TABLE_LEN*sigmoid(env.observation_space.low[1]),\n",
    "                    env.observation_space.low[2],\n",
    "                    Q_TABLE_LEN*sigmoid(env.observation_space.low[3])])\n",
    "\n",
    "discrete_os_win_size = (observation_high - observation_low) / DISCRETE_OS_SIZE\n",
    "\n",
    "# q_table = np.random.uniform(low=0, high=1,\n",
    "#                             size=(DISCRETE_OS_SIZE + [env.action_space.n]))\n",
    "\n",
    "q_table = np.zeros((DISCRETE_OS_SIZE + [env.action_space.n]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(150, 150, 150, 150, 2)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_table.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decay epsilon "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "epsilon = 1  # not a constant, qoing to be decayed\n",
    "START_EPSILON_DECAYING = 1\n",
    "END_EPSILON_DECAYING = EPISODES//2\n",
    "epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Help functions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_discrete_state (state):\n",
    "    discrete_state = (state - observation_low) // discrete_os_win_size\n",
    "    return tuple(discrete_state.astype(int))\n",
    "\n",
    "def take_epilon_greedy_action(state, epsilon):\n",
    "    discrete_state = get_discrete_state(state)\n",
    "    if np.random.random() < epsilon:\n",
    "        action = np.random.randint(0,env.action_space.n)\n",
    "    else:\n",
    "        action = np.argmax(q_table[discrete_state])\n",
    "    return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Rewards Recorder setup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "ep_rewards = []\n",
    "aggr_ep_rewards = {'ep':[],'avg':[],'min':[],'max':[]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Train the Agent "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode: 0\n",
      "episode: 1000\n",
      "episode: 2000\n",
      "episode: 3000\n",
      "episode: 4000\n",
      "episode: 5000\n",
      "episode: 6000\n",
      "episode: 7000\n",
      "episode: 8000\n",
      "episode: 9000\n",
      "episode: 10000\n",
      "episode: 11000\n",
      "episode: 12000\n",
      "episode: 13000\n",
      "episode: 14000\n",
      "episode: 15000\n",
      "episode: 16000\n",
      "episode: 17000\n",
      "episode: 18000\n",
      "episode: 19000\n",
      "episode: 20000\n",
      "episode: 21000\n",
      "episode: 22000\n",
      "episode: 23000\n",
      "episode: 24000\n",
      "episode: 25000\n",
      "episode: 26000\n",
      "episode: 27000\n",
      "episode: 28000\n",
      "episode: 29000\n",
      "episode: 30000\n",
      "episode: 31000\n",
      "episode: 32000\n",
      "episode: 33000\n",
      "episode: 34000\n",
      "episode: 35000\n",
      "episode: 36000\n",
      "episode: 37000\n",
      "episode: 38000\n",
      "episode: 39000\n",
      "episode: 40000\n",
      "episode: 41000\n",
      "episode: 42000\n",
      "episode: 43000\n",
      "episode: 44000\n",
      "episode: 45000\n",
      "episode: 46000\n",
      "episode: 47000\n",
      "episode: 48000\n",
      "episode: 49000\n"
     ]
    }
   ],
   "source": [
    "for episode in range(EPISODES):\n",
    "    # initiate reward every episode\n",
    "    ep_reward = 0\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        print(\"episode: {}\".format(episode))\n",
    "        render = True\n",
    "    else:\n",
    "        render = False\n",
    "\n",
    "    state = env.reset()\n",
    "    action = take_epilon_greedy_action(state, epsilon)\n",
    "    done = False\n",
    "\n",
    "    while not done:\n",
    "\n",
    "        next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "        ep_reward += reward\n",
    "        \n",
    "        next_action = take_epilon_greedy_action(state, epsilon)\n",
    "\n",
    "        # if render:\n",
    "        #     env.render()\n",
    "\n",
    "        if not done:\n",
    "\n",
    "            td_target = reward + DISCOUNT * q_table[get_discrete_state(next_state)][next_action]\n",
    "\n",
    "            q_table[get_discrete_state(state)][action] += LEARNING_RATE * (td_target - q_table[get_discrete_state(state)][action])\n",
    "\n",
    "\n",
    "\n",
    "        state = next_state\n",
    "        action = next_action\n",
    "\n",
    "    # Decaying is being done every episode if episode number is within decaying range\n",
    "    if END_EPSILON_DECAYING >= episode >= START_EPSILON_DECAYING:\n",
    "        epsilon -= epsilon_decay_value\n",
    "\n",
    "    # recoard aggrated rewards on each epsoide\n",
    "    ep_rewards.append(ep_reward)\n",
    "\n",
    "    # every SHOW_EVERY calculate average rewords\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        avg_reward = sum(ep_rewards[-SHOW_EVERY:]) / len(ep_rewards[-SHOW_EVERY:])\n",
    "        aggr_ep_rewards['ep'].append(episode)\n",
    "        aggr_ep_rewards['avg'].append(avg_reward)\n",
    "        aggr_ep_rewards['min'].append(min(ep_rewards[-SHOW_EVERY:]))\n",
    "        aggr_ep_rewards['max'].append(max(ep_rewards[-SHOW_EVERY:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Rewards')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEGCAYAAAB2EqL0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9bn48c+TnWxkD0sIYd9XA1URizvuWm0rte5LF63+2nrV1tt77e1tr13u7a3tbSsupVZEbd2tS9WqiIAQFlmDBAgQsidk3zPP749zEgKEZAKZTDJ53q/XvDLzPWfOPAdm5pnvcr5fUVWMMcaY7gT5OwBjjDEDgyUMY4wxXrGEYYwxxiuWMIwxxnjFEoYxxhivhPg7gFORlJSkGRkZ/g7DGGMGlA0bNpSqanJPnzegE0ZGRgZZWVn+DsMYYwYUEdl/Ms+zJiljjDFesYRhjDHGK5YwjDHGeGVA92F0prm5mby8PBoaGvwdis9ERESQlpZGaGiov0MxxgwiAZcw8vLyiImJISMjAxHxdzi9TlUpKysjLy+PMWPG+DscY8wg4rMmKREZJSIfiMgOEdkuIve65Qki8q6I7Hb/xrvlIiKPikiOiGwRkbkn87oNDQ0kJiYGZLIAEBESExMDugZljOmffNmH0QJ8X1WnAqcDd4nIVOBB4H1VnQC87z4GuBiY4N7uBP5wsi8cqMmiTaCfnzGmf/JZk5SqFgAF7v1qEdkJjASuBBa5u/0Z+BB4wC1/Wp351teKSJyIDHePY8xJU1Ve3/s6B6oO+DsUY3rN+PjxLM5Y3Kev2Sd9GCKSAcwBPgVSOySBQiDVvT8SONjhaXlu2VEJQ0TuxKmBkJ6e7rOYTWBQVX6V9Sue3vE0AILVzkxgWJyxOPAShohEAy8C/09Vqzo2p6iqikiPVnBS1aXAUoDMzExb/SkAbC/dTlRoFBlDM3r1uKrKL7N+yV92/IXrp1zPA/MesOY8Y06BTxOGiITiJIvlqvqSW1zU1tQkIsOBYrf8EDCqw9PT3LIB6aqrruLgwYM0NDRw77334vF42LNnD7/85S8BWLZsGVlZWfzud7/jJz/5Cc888wzJycmMGjWK0047jfvuu8/PZ9B3frDqBwyLHMbSC5f22jFVlV+s/wXP7HyGr0/5OvfPu9+ShTGnyGcJQ5xP55PATlX9nw6bXgNuAh5x/77aofxuEXkO+AJQear9Fz9+fTs78qtO5RDHmToiln+/fFq3+z311FMkJCRQX1/PvHnzeP/991mwYEF7wnj++ed56KGHWL9+PS+++CKfffYZzc3NzJ07l9NOO61XY+7PVJWCmgLK6stQ1V75UldVfr7+5yzfudyShTG9yJejpBYANwDnishm93YJTqK4QER2A+e7jwHeBPYCOcDjwLd9GJvPPfroo8yaNYvTTz+dgwcPsm/fPsaOHcvatWspKysjOzubBQsW8Mknn3DllVcSERFBTEwMl19+ub9D71NVTVU0tDZQ1VRFXk3eKR9PVXlk3SMs37mcG6feaMnCmF7ky1FSq+CEPYzndbK/Anf1Zgze1AR84cMPP+S9995jzZo1REZGsmjRIhoaGrjuuut44YUXmDx5MldffbV9kQFFdUXt97eXbWdUzKgu9u6aqvJf6/6LFdkruGnqTXw/8/v2b2xML7K5pHygsrKS+Ph4IiMjyc7OZu3atQBcffXVvPrqq6xYsYLrrrsOgAULFvD666/T0NBATU0Nb7zxhj9D73NFtUcSxo6yHad0rM0lm1mRvYIbpt5gycIYHwi4qUH6g8WLF/PHP/6RKVOmMGnSJE4//XQA4uPjmTJlCjt27GD+/PkAzJs3jyuuuIKZM2eSmprKjBkzGDp0qD/D71PFdc6Yh8SIRHaUnlrCWHVoFcESzLdmfcuShTE+YAnDB8LDw3nrrbc63dZZDeK+++7j4Ycfpq6ujrPPPntQdXoX1RUhCGennc17+987pY7vNflrmJk8k5iwmF6O0hgD1iTVL9x5553Mnj2buXPncs011zB37klNozUgFdcVkxCRwMzkmVQ3V3Ow+mD3T+pERUMF20q3ccaIM3o5QmNMG6th9APPPvusv0Pwm8K6QlKjUpmW6AxQ2FG2g/TYnl/Bv7ZwLYpy5ogzeztEY4zLahjGr4pqi0iJTGF83HhCg0LZXrb9pI6zJn8NMWEx7YnHGNP7LGEYvyquKyY1MpXQ4FAmxU86qZFSqsrq/NWcPvx0QoKs0myMr1jCMH5T31JPVVMVqZHO/JNTE6eys2wnHvX06Dj7qvZRWFtozVHG+JglDOM3bUNqU6OchDEtadpJdXyvPrQawDq8jfExSxh+8tprr/HII490v2MAa7tor2MNA3p+Ad/q/NVkxGYwMnpk7wZojDmKJQw/ueKKK3jwwQe73zGAtU0LkhKZAsC4uHGEBYWxvdT7ju+m1iayirKsdmFMH7CE4QO5ublMnjyZm2++mYkTJ3L99dfz3nvvsWDBAiZMmMC6detYtmwZd999NwA333wz99xzD2eeeSZjx47lb3/7m5/PoG+0JYy2GkZoUCiTEiaxo9z7Gsbm4s3Ut9Rb/4UxfSCwh5S89SAUbu3dYw6bARd335SUk5PDX//6V5566inmzZvHs88+y6pVq3jttdf42c9+xlVXXXXU/gUFBaxatYrs7GyuuOIKrr322t6Nux8qqi0iJjSGyNDI9rKpiVN5Y+8beNRDkHT/e2Z1/mpCgkKYP2y+L0M1xmA1DJ8ZM2YMM2bMICgoiGnTpnHeeechIsyYMYPc3Nzj9r/qqqsICgpi6tSpFBUVHX/AAFRcV9ze4d1mWuI0aptrvV5/e3X+amYnzz4q6RhjfCOwaxhe1AR8JTw8vP1+UFBQ++OgoCBaWlq63N+Z6T3wFdUVtTdHtWnr+N5etr3bJVvL6svYWb6Te+bc46sQjTEdWA3D+E1xXXF7h3ebcXHjCA8O92qk1NoCZ9p4678wpm9YwjB+0expprS+9LgmqZCgECYlTPJqipDV+auJC49jcsJkX4VpjOnAl2t6PwVcBhSr6nS37HlgkrtLHFChqrNFJAPYCexyt61V1W/6KjZfy8jIYNu2be2Ply1b1um2m2+++bjtADU1Nb4O0e/K6stQ9LgaBsDUhKm8vvf1Lju+VZU1+Ws4Y/gZBAcF+zpcYwy+rWEsAxZ3LFDVr6rqbFWdDbwIvNRh8562bQM5WRjvFNYWAhzXhwFOP0Ztcy37q/af8Pm7K3ZTUl9i118Y04d8ljBUdSVQ3tk2cVbI+Qqwwlevb/q3Y6/B6GhakjPjbFfNUmvy1wA2HYgxfclffRgLgSJV3d2hbIyIbBKRj0Rk4YmeKCJ3ikiWiGSVlJT4PlLjE+3zSHWSMMYOHUtEcESXHd+r81czbug4hkUN81mMxpij+SthLOHo2kUBkK6qc4DvAc+KSGxnT1TVpaqaqaqZycnJfRCq8YWi2iLCg8MZGn78+uXtHd8nmCKkoaWBDUUbrHZhTB/r84QhIiHAl4Dn28pUtVFVy9z7G4A9wMS+js30nbYhtSdav3tq4lSyy7Np9bQet+2d3HdobG204bTG9DF/1DDOB7JVNa+tQESSRSTYvT8WmADs9UNspo90dtFeR9MSp1HXUndUx3d1UzUPr36Yf/3kXxkfN555w+b1RajGGJfPEoaIrADWAJNEJE9EbnM3Xcfxnd1nA1tEZDPwN+Cbqtpph7kJDEV1RZ0OqW3T8YpvgFWHVnH1q1fzcs7L3Dr9Vp677DkiQiL6JFZjjMNn12Go6pITlN/cSdmLOMNszSDgUU+n80h1NGboGIaEDOHTgk9ZV7iOV3JeYdzQcfx60a+ZkTyjD6M1xrSxK719wJvpzdetW8cZZ5zBnDlzOPPMM9m1y7lm8de//jW33norAFu3bmX69OnU1dX583R63eGGwzR7mrtskgoJCmFS/CRe3fMqr+95nTtm3MELl79gycIYPwroyQd/vu7nZJdn9+oxJydM5oH5D3S7X3fTmz/99NN8/PHHhISE8N577/HDH/6QF198kXvvvZdFixbx8ssv89Of/pTHHnuMyMjAmom1qyG1HV0y9hKCJIj7593ffm2GMcZ/Ajph+FPb9OZAp9ObV1ZWctNNN7F7925EhObmZsCZzXbZsmXMnDmTb3zjGyxYsMCfp+ETXV2019GSyUtYMrnTlk1jjB8EdMLwpibgK91Nb/6jH/2Ic845h5dffpnc3FwWLVrUvv/u3buJjo4mPz+/r8PuE201jK46vY0x/Y/1YfhJZWUlI0eOBI6efLCyspJ77rmHlStXUlZWFpDLtRbWFhIswSQNSfJ3KMaYHrCE4Sf3338/P/jBD5gzZ85RCyp997vf5a677mLixIk8+eSTPPjggxQXF/sx0t5XVFdE0pAkm2XWmAFGBvLqbpmZmZqVlXVU2c6dO5kyZYqfIuo7A/k87/jHHdQ117H80uX+DsWYQUlENqhqZk+fZzUM0+eK6oq6vAbDGNM/WcIwvUpV2VKypct1yTtbmtUY0/8FZMIYyM1s3ujP57emYA3Xv3k9Hxz8oNPtNU011DbXdjuk1hjT/wRcwoiIiKCsrKxff6meClWlrKyMiIj+OY9SVqHTp/T2vrc73W5Dao0ZuALuOoy0tDTy8vII5MWVIiIiSEtL83cYndpUvAmAD/M+pL6lniEhQ47aXlh34qVZjTH9W8AljNDQUMaMGePvMAal5tZmtpVuY3LCZLLLs1l1aBUXjL7gqH2Kat2rvK3T25gBJ+CapIz/7CzfSUNrA7dOv5WEiATeyX3nuH2sScqYgcsShuk1bc1RmamZXDD6AlbmraS+pf6ofYrqiogPjyc8OLyzQxhj+jFLGKbXbCreRFp0GsmRyVyUcRH1LfV8nPfxUfvYkFpjBi5LGKZXqCqbijcxJ2UOAHNT5pIYkXhcs5RdtGfMwOXLJVqfEpFiEdnWoexhETkkIpvd2yUdtv1ARHJEZJeIXOSruIxvHKw+SHlDOXNSnYQRHBTM+aPPZ2XeSuqajywAVVTb9Vrexpj+y5c1jGXA4k7Kf62qs93bmwAiMhVnre9p7nN+LyI2M90AsrF4IwBzkue0l12UcRENrQ2sPLQSgMbWRg43HrYmKWMGKJ8lDFVdCZR7ufuVwHOq2qiq+4AcYL6vYgt020u388Xnv0h+Td+tp7G5eDMxYTGMjRvbXjY3ZS5JQ5L4R+4/AO9X2jPG9E/+6MO4W0S2uE1W8W7ZSOBgh33y3LLjiMidIpIlIlmBfHHeqfj40MeUN5TzacGnffaabf0XQXLkLRUcFMz56efzcd7H1DXXWcIwZoDr64TxB2AcMBsoAP67pwdQ1aWqmqmqmcnJyb0dX0DYXrodgC2lW/rk9SoaKthbube9w7uj9mapvJV20Z4xA1yfJgxVLVLVVlX1AI9zpNnpEDCqw65pbpnpIVVlW5kzzmBrydY+ec3NJZsBOk0Yc1LmkDwkmXdy37GL9owZ4Po0YYjI8A4PrwbaRlC9BlwnIuEiMgaYAKzry9gCRVFdEaX1pSREJLC7YvdRI5R8ZWPxRkKCQpiWOO24bcFBwVww+gI+PvQx+6r2ERkSSXRotM9jMsb0Pl8Oq10BrAEmiUieiNwG/EJEtorIFuAc4LsAqrodeAHYAbwN3KWqrb6KLZBtK3Vy8JcnfhmPethRtsPnr7m5eDNTE6cSEdL5DLoXZlxIY2sjb+17i9SoVETE5zEZY3qfL0dJLVHV4aoaqqppqvqkqt6gqjNUdaaqXqGqBR32/6mqjlPVSar6lq/iCnTbSrcRIiFcO/FaALaW+rZZqrG1kW2l25ibMveE+7Q1S9W31FtzlDEDmF3pHWC2lW1jQvwEhkUNIy06jS0lvu343lG2g2ZPM7NTZp9wnyAJ4sKMCwEbIWXMQGYJI4B41MOO0h1MT5oOwMzkmT4fKdU24WBnHd4dXZThXLxvCcOYgcsSRgDZX7Wf6ubqoxJGcV1x+3BWX9hUtImM2AwSIhK63G9W8ixumXZLe+Iwxgw8ljACSFuHd9topRlJMwDf9WOoKptLNnfZHNUmSIL4Xub3mJQwySexGGN8zxJGANletp0hIUMYFzcOgMkJkwkNCvVZs9S+qn1UNFZ02eFtjAkcljACyLbSbUxJmEJIkLPyblhwGJMTJvvsAr5NRU7/hTc1DGPMwGcJI0A0e5rJLs9mWtLRF8/NSJrB9rLttHhaev01NxVvIj48nozYjF4/tjGm/7GEESByDufQ2NrI9MTpR5XPSJ5BfUs9eyr29PprbirexOyU2XYhnjGDhCWMANE2f1RbR3ebWUmzgJOfiHB94Xoe3fgor+95nR1lO9rX6C6tL+VA9QHrvzBmEAnxdwCmd2wv3c7Q8KGkxaQdVZ4Wk0Z8eDxbS7by5Ylf7vFxf7n+l+ws39n+WBBGRo8kPsKZmd76L4wZPCxhBIitpVuZljjtuOYhEWF60vSTGlpbWFvIzvKdfGfOdzg//Xz2VO5hT4V7q9zDlIQpTE2c2lunYIzp5yxhBIC2PopFoxZ1un1G8gxWHVpFTVMN0WHezxT70cGPADg//XzGxo1lbNxYLhh9QW+EbIwZgKwPIwBkl2fTqq3HdXi3mZk0E+XIOhne+jDvQ0bFjGLM0DG9EaYxZoCzhBEA2q7wbpsS5Fht5T25HqOuuY51BetYNGqRjYIyxgCWMALC1tKtpESmkBzZ+ZK1Q8OHkhGb0aORUmvy19DkaWJR2qJeitIYM9BZwggA20u3Hzec9lgzk2eytWQrqurVMT/M+5CY0BjmpHY9C60xZvDw5Yp7T4lIsYhs61D2SxHJFpEtIvKyiMS55RkiUi8im93bH30VV6CpbKzkQPWBEzZHtZmRNIOyhjIKagu63A+cadJX5q3krJFnERoU2luhGmMGOF/WMJYBi48pexeYrqozgc+BH3TYtkdVZ7u3b/owroCyvWw7QKfraXc0I9mpgXjTLLW1dCvlDeUnHHVljBmcfLlE60qg/Jiyf6hq26RGa4G0455oeqR9SvOkrhPGxPiJhAeHe7UC30cHPyJYglkwckGvxGiMCQz+7MO4Fei4dvcYEdkkIh+JyMITPUlE7hSRLBHJKikp8X2U/dy20m1kxGYQGxbb5X6hQaFMSZji1UipDw5+wNzUuQwNH9pbYRpjAoBfEoaIPAS0AMvdogIgXVXnAN8DnhWRTr8BVXWpqmaqamZycuejgvq7QzWHvO587s720u3d1i7azEiewc7ynTR7mruMLacihy+mfbFX4jPGBI4+TxgicjNwGXC9ut+aqtqoqmXu/Q3AHmBiX8fWF/ZV7uOSly7hjb1vnPKximqLKK4vPuEFe8eamTyTxtZGPj/8+Qn3+fDghwDWf2GMOY5XCUNExolIuHt/kYjc0zbCqSdEZDFwP3CFqtZ1KE8WkWD3/lhgArC3p8cfCNbkr8GjHt7JfeeUj9V25XZ3I6TazE52JgpcsXPFCWs4Hx38iDFDxzA6dvQpx2eMCSze1jBeBFpFZDywFBgFPNvVE0RkBbAGmCQieSJyG/A7IAZ495jhs2cDW0RkM/A34JuqWt7pgQe4rKIswEkctc21p3Ss9YXrCZEQJidM9mr/YVHDuGPGHby651We3PbkcdtrmmpYX7TeLtYzxnTK28kHParaIiJXA79V1d+KyKaunqCqSzopPv5bytn3RZykFNA86iGrMIuM2Axyq3L5+NDHLM44duSxd+qa63gt5zXOH30+ESERXj/vO3O+Q15NHr/Z+BvSotNYPObI63+S/wktnha+OMr6L4wxx/O2htEsIkuAm4C2xne7oquHcipyONx4mFun30pCRAL/3P/Pkz7Wa3teo7q5muunXN+j54kIP1nwE+akzOGhVQ+xuXhz+7aPDn5EXHgcs5JnnXRcxpjA5W3CuAU4A/ipqu4TkTHAX3wXVmBaX7gegC8M/wKLRi1i5aGVNLU29fg4HvXwbPazTEucdlJf7uHB4fzmnN8wLGoY9/zzHg5WHaTV08rHhz5m4ciFhATZrPfGmON5lTBUdYeq3qOqK9zH+1T1574NLfBkFWYxMnokI6JHcF76edQ21/Jpwac9Ps7a/LXsq9zH9VOuP+mZZOMj4vn9+b/Hg4dvv/9tPsr7iIrGCmuOMsacUJcJQ0S2uvM+dXrrqyADgUc9rC9az7xh8wCnlhEVGsX7B97v8bGWZy8nMSKRizIuOqWYRseO5tFzHuVQzSHu++g+QoJCWDDCru42xnSuuxrGZcDlwNvu7Xr39hbwpm9DCyy7D++msrGyPWGEB4ezcORCPjj4Aa2eVq+Ps79qPyvzVvKVSV8hLDjslOOamzqX/1zwnzR7mslMzezRinzGmMGly8ZqVd0PICIXuFdht3lARDYCD/oyuEDSNpx2Xuq89rLz0s/j7dy32VyymdNST/PqOCuyVxASFMJXJn2l12K7ZOwlxIbHMipmVK8d0xgTeLzt9BYRWdDhwZk9eK4B1hWsIy06jeHRw9vLFqYtJDQo1OtmqZqmGl7JeYXFGYtJGpLUq/GdNfIsu1jPGNMlb7/0bwV+LyK5IpIL/N4tM17wqIesoqz25qg2UaFRnDHiDN7f/75Xc0u9uudVaptrezyU1hhjekO3CUNEgoDxqjoLmAXMctes2Ojz6ALE7sO7qWqqOi5hgNMslV+bT3Z5dpfH8KiHZ3c+y6zkWV5PBWKMMb2p24Shqh6c+Z9Q1UpVrfR5VAFmXeE6gE4TxqJRiwiSoG6bpVYdWsWB6gNWuzDG+I23TVLvich9IjJKRBLabj6NLICsL1zPqJhRDIsadty2hIgE5qbM7TZhLN+5nJQhKZw/+nxfhWmMMV3yNmF8FbgLWAlscG9ZvgoqkLR6WskqymL+sPkn3Oe89PPIqchhf9X+TrfvrdjL6vzVfHXyV22NbWOM33h7pfeYTm5jfR1cIPj88OdUN1WTOSzzhPucm34uQKe1jIKaAn6R9QvCgsK4duK1PovTGGO64/WkQSIyHZgKtE+NqqpP+yKoQNI2f1TH6y+ONSJ6BFMTp/L+gfe5dboz+Cy/Jp8ntj7ByzkvA3DPnHtIiLBWQGOM/3iVMETk34FFOAnjTeBiYBVgCaMb6wvXMzp2NKlRqV3ud176efx202/ZXLyZV3Je4dU9ryII10y4htum33bU9RvGGOMP3tYwrsUZUrtJVW8RkVTgGd+FFRhaPa1sKNrAhRkXdrtvW8K44a0bCA0K5doJ13LbjNs67Sg3xhh/8DZh1KuqR0RaRCQWKMZZdc90YdfhXVQ3V3c6nPZYY4eO5UsTvkREcAS3TL/FEoUxpt/xNmFkuWt4P44zQqoGZ/nVLonIUzgTGBar6nS3LAF4HsgAcoGvqOphcebp/g1wCVAH3DzQLw5s77/wImGICD8+88e+DskYY06at6Okvq2qFar6R+AC4CZVvcWLpy4Djl2D9EHgfVWdALzPkQkMLwYmuLc7gT94E1t/tr5wPRmxGaREpvg7FGOMOWVeJQwR+YuI3CEik1U1V1W9WgtDVVcC5ccUXwn82b3/Z+CqDuVPq2MtECciA7ant63/oqvhtMYYM5B4e+HeU8Bw4LcisldEXhSRe0/yNVNVtcC9Xwi0DR8aCRzssF+eW3YUEblTRLJEJKukpOQkQ/C97PJsapprurxgzxhjBhJvm6Q+AH4K/AinHyMT+Napvrg6U7R2P03r0c9ZqqqZqpqZnJx8qiEcZ2XeSvZU7Dnl47y0+yVCgkK86r8wxpiBwNvrMN4HonA6uj8G5qlq8Um+ZpGIDFfVArfJqe04hzh65FWaW9Znmj3NfOef3yFIgvjmzG9y64xbT2oqjvyafF7KeYkvjf9Sr69bYYwx/uJtk9QWoAmYDswEpovIkJN8zdeAm9z7NwGvdii/URynA5Udmq76RFVjFR71kBqZyu82/47r/349u8p39fg4T2x9AkG4Y+YdPojSGDPQeTxKdmEVyz7Zxzf/soHLfvsxq3aX+jusbnlVw1DV7wKISAxwM/AnYBgQ3tXzRGQFzhXiSSKSB/w78AjwgojcBuwH2tYafRNnSG0OzrBab0Zh9arKJmfm9nvn3ktoUCg/WfsTrnvjOu6ceSe3z7id0ODuaxv5Nfm8nPMy10y4xq6lMMYAoKp8XlTD6j2lrN1bxrp95RyuawYgLX4IInDLsnX86suzuHL2cV23/Ya3TVJ3AwuB03CunXgKp2mqS6q65ASbzutkX8WZEddvKhudhDE0bChnjjyTzNRM/mvdf/H7z37Pewfe42dn/YxJCZO6PMbSLUsRhNtn3N4XIRtj+qm8w3Wszinjkz2lfJJTRmlNIwCjEoZw/pRUvjA2kS+MSWBUQiSV9c3c+XQW9z63meKqRu44u3/O7erthXsRwP8AG1S1xYfx+FV7wggfCkBcRBw/P/vnXJRxET9Z+xNu/8ftLL9kOemx6Z0+P686j1dzXuXaidda7cKYQaKl1cOB8jpyimvYXVzDnuIaNh44TG5ZHQBJ0eEsGJ/IgvFJnDkukbT4yOOOMXRIKH++dT7fe2EzP31zJ0VVDfzwkikEBUlfn06XvG2S+pWInAXcAPxJRJKBaFXd59Po+lhbwogNjz2q/Nz0cxkfN57r37yeb7//bZ65+BniIuKOe/4TW58gSIKsdmEMUFjZwOdF1QQHydE3EYYNjSA1NqL7g/RT+RX1/N8HOWTlHmZfaS1NrZ72bcNiI5g2IpYbz8hgwfgkJqZG40xk0bWI0GB+u2QuKTE7eGLVPoqrG/nll2cSHhLsy1PpkZ7MVpsJTMLpvwjFmXxwge9C63ttCSMu/PhkkB6bzqPnPsrt79zOvR/cy+MXPk5YcFj79rbaxVcmfaXbmWmNCXTbDlWy5PG1VDd03iARJHDx9OHcvnAMc9Lj+zi6k1dZ38wfPtzDnz7ZhyqcNSGJRZOSGZ8SzfiUaMalRBMbcfKLnAUHCf9++VRSYyP4+dvZlNU28siXZjIq4fhaiT942yR1NTAH2AigqvluB3hAqWyqJFiCiQ6N7nT7nJQ5/PSsn/IvK/+FH33yIx5Z+Ej7L4elW5YSJEHcNuO2vgzZmH5nV2E1Nzz5KbERofzh+tMIDSmxr0kAAB+iSURBVBZaVWn1HLll7T/M8rX7+fvWAjJHx3P7wrFcMDWVYLcJRlXJO1zPxgOH2bj/MLuLazh3cgpL5qcTFe71Mj69prGllb+s2c/vPsihsr6Zq2eP5HsXTuy0eelUiQjfWjSOlJhwHnhxCwt/8QETUqI5Z3IKiyYlMy8jgdBgbwe49i5v/+WbVFVFRAFEJMqHMflNZWMlsWGxXVYfF49ZTF5NHr/Z+BtGxYzi7jl3c7DqIK/teY3rJl9n80aZQW1PSQ3XP7GWsJAgnr3jC4xO7Pyr4rwpqdx9znheyDrIk6v28c1nNpCRGMllM0ewp6SGDfsPU1ztdBJHhgUzMm4I//n3nfzugxxuPjODm8/MIC4yrNNj91R9UyufF1WTW1aLR5UgEUSEIIEgEcprm/jjR3vIO1zPwglJPHjxZKaNGNorr92Va05LIzMjnnd3FPHhrhL+9Mk+lq7cS0x4CGdNSOLyWSO4ZEbfzp7kbcJ4QUQew5nf6Q7gVuAJ34XlH5WNle0d3l25bfptHKw+yGNbHiMtJo0NRRsICQrhtulWuzCD14GyOq5//FMAlt9++gmTRZuo8BBuWTCGG04fzTvbi3j847387oMcRiUM4cxxiZw2Op456fFMHhZDSHAQGw8c5vcf7OF/39vN0pV7+dr8dG5fOJZhQ73vCzlc28TGA4fZWVDFzsJqdhZUkVtai6eb+SamDo/lL7fNYOGE3p9doiujE6O4feFYbl84lprGFj7JKeXDXcV8kF1CbERonycMcUazerGjyAXAhYAA76jqu74MzBuZmZmalZXVa8e78x93UttSy/JLlne7b7OnmW+/922yCrNQlCWTl/DA/Ad6LRZjBpJDFfV85Y9rqG1q4bk7T2fysNjun3QMVaWuqbXbJqddhdX88aM9vPZZPsEinD4ukdPS4zltdDyz0+OI7vB8j0fZnl/FB7uK+XBXMZsPVrQnh/SESCYPi2HK8FimDI9hXHI0IcFBeFRRVVTBo05/y7jk6H41Ysnbf6sTEZENqtrjmVG9ThjHvFgQsERVu/9m9aHeThhffeOrJEYk8vvzf+/V/tVN1dz41o3kVefx5pfeJDmyb399GNPbXv8sn7+s3c+/XDSJeRnerSFfVNXAVx9bQ1ltE8/efjoz0nzfXANwsLyOP32Sy+o9pewqqkbdL/fJw2I5bXQ8dU2tfPR5Sfv1D7PShvLFSSmcNT6JKcNjiDmFzumB7mQTRpfpyV1d7y6cWWNfA951H98HfAb4NWH0tsrGSsYNHef1/jFhMfz54j9TVl9mycIMeP/MLuK7z28G4CuPreGmMzK4f/EkIsM6/5rweJR/7CjkkbeyKa5u5C+3faHPkgXAqIRI/u3yqQBUNTSz+UAFWfudTvKXNuYREhzE2ROTOWdSMmdPTCYpusuJKYwXuqvP/AU4jDPp4O3AD3GapK5S1c0+jq3PVTVWedWH0VFsWCyxYT2vfhvTn2TllvPt5RuZPDyGJ2+ax/99kMOy1bn8M7uYn18zkzPGJbbv6/Eob20r5Lf/3E12YTVjk6L4863zOW20/4bHxkaEcvZEJzEAtHoUgX7VjBQIuksYY1V1BoCIPAEUAOmq2uDzyPpYi6eF6ubq4y7aMybQ7Syo4tZl6xkxdAjLbplPUnQ4/3HldC6dMZz7X9zCksfX8vXT07l/8WQ+2lXCb/+5m8+LahibHMX/fnU2l88a0T4ctr/ob/EEiu4SRnPbHVVtFZG8QEwWAFVNVYAzj5Qxg8WBsjpufGodkWEhPH3b/KOabb4wNpG37z2bX/1jF099so8X1ufR1OphfEo0v7luNpfN7H+JwvhWdwljlohUufcFGOI+Fpz5AgPm5/ix80gZE+iKqxv4+pOf0tzq4dlvnNHpRWhDwoL50WVTuWTGcJZ/up9zJqVwyYzhligGqS4Thqr2n0lMfMwShhlMKuubuemp9ZRUN7L8ji8wIbXriRtOGx3v1z4K0z/0/TX2/VRbk1Rn80gZEygq6ppY/ukBnl6TS3ltE0/cNI+5A2guJ+NfljBcHdfCMCbQ7C2p4alP9vHihkPUN7eycEIS3zl3AvPHeHethTFgCaPdiaY2N2agamhuZc3eMp5Zs5/3s4sJCw7iqjkjuPWsMSd1JbYxfZ4wRGQS8HyHorHAvwFxwB1AiVv+Q1V9s6/iqmisQBBiwgJuEl4TQBpbWgkLDjrhBJmHKurduYaK+SSnjPrmVhKjwrj3vAl8/fTRJMfYxWvm5PV5wlDVXcBsABEJBg4BL+Os4f1rVf1VX8cE7ky14bEEiX+mDTamO3/bkMcPX9qKR5X4qDASo8JIjA4jISqc6PAQNh04THZhNeCsE/3lzDTOmZTCGeMSiQgdNONXjA/5u0nqPGCPqu73ZkUqX6psqrT+C9NvrdpdyoMvbmFOehzzMhIor22irLaJ8tomth2qpKKuiSnDY3nokimcMzmZccnerfJmTE/4O2FcB6zo8PhuEbkRyAK+r6qHj32CiNwJ3AmQnt752tono6qxykZImX4pu7CKbz2zgfEp0Tx587xTWtHNmFPht/YXEQkDrgD+6hb9ARiH01xVAPx3Z89T1aWqmqmqmcnJvTfhX1uTlDH9SVFVA7f+aT2R4cE8ZcnC+Jk/G+wvBjaqahGAqhapaquqeoDHgfl9GUxlk3eLJxnTV2oaW7jlT+uprG/mqZvnMSJuiL9DMoOcP5ukltChOUpEhqtqgfvwamBbXwZT0VhhfRimzxwoq+N7L2xmf3kdi6cN49KZw5mXkdA+5UZLq4e7lm9kV1E1T96U2SdLghrTHb8kDHdN8AuAb3Qo/oWIzAYUyD1mm0+1elqpbqq2GobpE29tLeD+v21BBE4fm8hfNxzkL2v3kxITziUzhnPpzOG8tPEQH31ewn99aQaLJtk68aZ/8EvCUNVaIPGYshv8EQs4K+eBzSNlfKuhuZWfvbmTp9fsZ9aoOH63ZA6jEiKpa2rh/Z3FvLEln2fXHWDZ6lwAvr1oHEvm997ADmNOlb9HSfULlU028aDxrdzSWu56diPb86u47awxPLB4MmEhThdiZFgIl88aweWzRlDT2ML7O4soq2ni5jMz/Bu0McewhIHNI2V8p7SmkXd3FPHTv+8kOEh4/MZMLpiaesL9o8NDuHL2yD6M0BjvWcLA6fAGq2GYU1dZ38y6feWs3lPKmj1l7Vdez0mP47dL5nS65oQxA4UlDGwtDHNqGppbee2zfFasO8BnByvwKISHBDEvI4F/uWgEZ4xLZFZanC06ZAY8SxjY8qzm5BRWNvDM2v08u+4A5bVNTEyN5u5zJ3DmuETmpMcRHmLzN5nAYgkDp4ZhM9Uab208cJg/fZLLW1sLaFXl/Cmp3LIggzPGJtr8TSagWcLASRgxYTEEB9kvQnNiqsov3tnFHz7cQ0xECDefmcGNZ2SQnmj9EmZwsISBTQtiuqeq/Pj1HSxbncvXvpDOQ5dMISrcPj5mcLF3PDYtiOmax6M89MpWVqw7yO1njeGhS6dY05MZlCxh4ExtbjUM05mWVg/3/20LL206xN3njOf7F060ZGEGLVteDpva3HSuudXDvc9t5qVNh7jvwoncd9EkSxZmULMaBrbanjleXVML96zYxHs7i/nXS6dw+8Kx/g7JGL8b9AnDox5ntb0IW21vMFJViqsb2VlQxc6CavdvFXtLa2n1KD+5ajo3nD7a32Ea0y8M+oRR3VSNolbDGIRKaxr51jMbWJ97ZCXgkXFDmDI8houmDeOsCUmcPjaxiyMYM7gM+oRh04IMTntLarj5T+sprm7ggcWTmZMex5RhsQyNtCVQjTkRSxiWMAad9bnl3PF0FsEirLjjdOakx/s7JGMGBEsY7loYsWE2SmoweP2zfL7/189Iix/Cspvn21XaxvSA3xKGiOQC1UAr0KKqmSKSADwPZOAs0/oVVT18omP0BqthDA6qyh8/2svP385mfkYCS288jbjIMH+HZcyA4u8axjmqWtrh8YPA+6r6iIg86D5+wJcBtCWMuHAbJRUIVJWqhhZKqhsoqmqk2P372cEK3tpWyOWzRvDLa2cSEWrzhhnTU/5OGMe6Eljk3v8z8CG+Thhuk5TNVDtweTzKp/vKeXFjHu9sL6S6oeW4faLDQ7j7nPF874KJBNm6FMacFH8mDAX+ISIKPKaqS4FUVS1wtxcCx61lKSJ3AncCpKenn3IQlY2VxITGEBLU33Kn6c7+slpe3HiIlzbmkXe4nujwEC6aNozJw2JIiQ0nJSaC1NhwUmIjiLaJAo05Zf78FJ2lqodEJAV4V0SyO25UVXWTCceULwWWAmRmZh63vadsWpCBpaXVw9vbC3l69X7W5ZYjAmeNT+K+Cydx0bRhDAmzpiZjfMVvCUNVD7l/i0XkZWA+UCQiw1W1QESGA8W+jqOy0aY2Hwjqm1r524aDPP7xPg6U1zEmKYr7F0/i6jkjGT50iL/DM2ZQ8EvCEJEoIEhVq937FwL/AbwG3AQ84v591dex2DxS/dvh2ib+snY/y1bnUl7bxJz0OB66dAoXTEm1vghj+pi/ahipwMvuzJ8hwLOq+raIrAdeEJHbgP3AV3wdSFVjFSOjRvr6ZUwPeDzK+txyXtl8iFc25VPf3Mp5k1P4xhfHMS8j3maMNcZP/JIwVHUvMKuT8jLgvL6MpaKxwvow+omc4mpe3uQkiUMV9USGBXPpzOHcsXAsk4bZKDZj/G1QDx3xqIeqJls8yZ88HuWvGw7yzNoDbD1USZDAwgnJ/MtFk7hwWiqRYYP6LWpMvzKoP401zTV41GN9GH6SU1zDD17awvrcw0wbEcu/XTaVy2YNJyUmwt+hGWM6MagThk0L4h9NLR4e+2gPv/1nDkPCgvnVl2dxzdyR1jdhTD83qBNGVWMVYAmjL312sIIHXtxCdmE1l84czsOXTyM5JtzfYRljvDCoE4bVMHzL41EKqhrYX1bLgbI6Psur4Pn1B0mOCefxGzO5YOpxF/IbY/qxQZ0wKhorAEsYvaGkupEteRV8llfJ9kOV5JbVcrC8nqZWT/s+ocHCkvnpPHDxZGIjbKEiYwaaQZ0w2iYetE7vnvu8qJp/Zhc7SeJgJYcq6gEIEhiXHM34lGjOn5JKemIkoxOiGJ0YyfChEYQEB/k5cmPMyRrcCcNtkrLrMLyTW1rLG1vyef2zAnYVVQMwKmEIc9LjuPnMDGamDWX6yKFE2UR/xgSkQf3JrmysJCo0itAgax45kYPldbyzvZDXP8vnszwnwc7LiOc/rpzG4mnDSIm1IbDGDBaDOmFUNVVZc9QxqhqaWbOnjI93l7Bqdym5ZXUAzBg5lIcumcKlM4czIs4m+zNmMBrUCcNmqnVGMm3Lr+SD7BJW7i5h88EKWj1KZFgwZ4xN5OYzM1g0KYWMpCh/h2qM8bNBnTAqGisGZcKoamhm1e5S/pldzIe7SiitaUTEqUV864vjOGtCEnPT4wkLsQ5qY8wRgzphVDZWMixqmL/D8ClVJe9wPVvyKtmSV8GmgxVs3H+YFo8SGxHC2ROTOXdyCl+cmExitF1AZ4w5sUGdMAKtD6OhuZV9pbXsLq7h88JqthyqZGteBYfrmgEICw5i8vAYbl84lnMnpzA3Pc6GuRpjvDZoE4aqDtg+jFaPsq+0hu35VWQXVrO7qIac4moOlNfhcRetDQ4SJqbGcOHUYcxIG8qstDgmDosmPMSWMDXGnJxBmzBqm2tp1dZ+nTBUleLqRvaX1fF5UTU7CqrYkV9FdmEVDc3OFdQhQcKYpCimjojlitkjGZ8SzYSUaMYkRRERasnBGNN7+jxhiMgo4GmcVfcUWKqqvxGRh4E7gBJ31x+q6pu+iqPtKu/YMP9etNc239K+klr2ltaQW1rHgfI6DpTXcqC8rj0xAMRGhDB1RCxfmz+aqSNimTYilnHJ0dY5bYzpE/6oYbQA31fVjSISA2wQkXfdbb9W1V/1RRBt80jFhcf1xcvR0NzKnpIacopr2F1Uw97SGvaW1JJbVntUUhgSGkx6QiSjE6M4e0IyoxMjSU+MYmxSFGnxQ2wKcGOM3/R5wlDVAqDAvV8tIjuBPl9U21cz1Xo8yoHyOrYeqmRHQdUJ+xfSEyIZmxTFWeOTGJMcxdikaMYmR5ESE25JwRjTL/m1D0NEMoA5wKfAAuBuEbkRyMKphRzu5Dl3AncCpKenn/Rrn+paGKpKWW0Thw7Xs7e0hm2Hqth2qJId+VVUN7YAzuysY5KimDZiKFfOHsnE1BgmpEaTkRhlzUjGmAHHbwlDRKKBF4H/p6pVIvIH4Cc4/Ro/Af4buPXY56nqUmApQGZmpp7s63tTw2hsaWV/WR173f6Fg+V15B2u51BFPfkV9Uc1JYWHBDFleCxXzhnBjJFDmTZiKBNTYywxGGMChl8ShoiE4iSL5ar6EoCqFnXY/jjwhi9j6KzTW1V5/OO9fJJTxr7SWvIOH2lGAkiMCmNk/BAmpcZw7qQURsYPYWTcEEYnRjEuOcquaTDGBDR/jJIS4Elgp6r+T4fy4W7/BsDVwDZfxlHZWMmQkCGEBYe1l7286RA/ezObSakxzBoVx9VzRjLW7V/ISIokxhb9McYMYv6oYSwAbgC2ishmt+yHwBIRmY3TJJULfMOXQVQ0Vhw1Qqq0ppH/eGMHp42O56/fOIOgIOt4NsaYjvwxSmoV0Nm3sc+uuehMVWPVUf0XD7+2nbrGVn5+zQxLFsYY04lB2+he2VTZPo/UezuKeGNLAd85dzzjU2L8HJkxxvRPgzdhNFYSGx5LVUMz//rKNiYPi+EbXxzn77CMMabfGrRzSbVNPPjIW9kUVzfw2A2n2RBYY4zpwqBMGG0z1dbWhfHCpwe4Y+EYZo3qmylCjDFmoBqUP6nrWupo0RY+2llLekIk37tgkr9DMsaYfm9Q1jDarvIuqw7h6S/PYEiYTQNujDHdGZQ1jO2fvQfArUn5nDk+yc/RGGPMwDAoE8ao0ZlcUxfL14pehDe+B63N/g7JGGP6vUGZMCanTeXhb6wk4wt3QdaT8MyXoK7c32EZY0y/Nij7MAAICoYL/gOSp8Dr98AT58GS5yDZOsADVmszlO2Bkmwo2QUlO52/iPP/njLF+Zs8BRLGQHAXc4c1VDrPLXaPUZINZbtPXFtNnQZTr4LJl0JkwgmOWQWfvw3bX4HSXZA4/kg8yZOcW1hUz8/b0wqHc50YO8ZbW9L5/iHhkDTRfc3J7utPhPBevKhVFWqKOsTj/q0uhISxR/9fJE+EiJNYhsDjgcqDRx+/eCc0VnU4P/d1kiZCWOSJj9Xc4Pz/Fme77x/3PdRUc/L/Bh0FhTjvubZ4UqY4//Yneq/4iaie9AzhfpeZmalZWVmnfqCD6+C5r0FLI1z7J5hw/qkfs7epOh/wkuyj37T1FZA0wXlzpUx2/iaMg5Cw7o/ZE7WlR16z7fUr83Cm/uoFkUlH4m+7DU0DEaf2d+xrVxzo2WurB6rywdPiFgjEZzgfTlX3mPuP7B8UCrHDQTqphDfXO192bUKGOF9qSRMhdMjx+3taIXeVc/ygEBjzRZh6JUy53Pnhsust2PEq5LwPrY0QMxxGzIXD+6B0N3g6JKGYET37v1V1voRbG4+UDR3lnHfMcOff91hNtVDyOZR+fvTzYoY7yaQ31B92km6bIfHOl2VMqpPUSz+HloYj26OHQWiE98dXdd6zzbVHHyN5kpN8SndDWU6Hf1uB2BGd/0hobYHqfOc9BCDBbjKf6MTdG1qanIR0bBKKTILw6M6fM+EiuOQXJ/VyIrJBVTN7/DxLGK6Kg/DcEijaDokTOv8g+Ysq1BY7H7I24UOdL9iIOOeNVr6P9i/QoBCIS4fgXkoataVQV9rhtWOdD17caOcL71SpQnWB82GpLT5SHhbtfAF3/CUcFu28dnyGc549ETvySGJNnHD8L8qmWueLqu0XeFV+58cJDnW/MKZ0+HfopnVXFQo2O7WHHa84v/gl2Pn3a21yYpt6pXNLm3/keK0tTuJo+yVevhe0tWfnHZXs/mLtYU3h2JpJ+d4OCfcUhUW7PwzcX9NRyUd/5jytzo+Cth8KZTk972scknDkR0jSxON/rbc2O+fU9kPkcG7n/7YS5Pwft8Xqix9kbVSh6tCRH0bHJs6O0ubB/DtO6mUsYfSGplr44GdONba/afsF1tZMEDPs6A9Yc73zq6mt+l2+r+dfLCcSMfToX/6xI3yXUI+tTTTXdl7rGMhUoeAzp1bhaYYpV8DIzO6TjjG9xBKGMcYYr5xswrCfNMYYY7xiCcMYY4xXLGEYY4zxSr9LGCKyWER2iUiOiDzo73iMMcY4+lXCEJFg4P+Ai4GpOOt8T/VvVMYYY6CfJQxgPpCjqntVtQl4DrjSzzEZY4yh/yWMkUDHiyDy3LJ2InKniGSJSFZJyQmmNjDGGNPr+lvC6JaqLlXVTFXNTE5O9nc4xhgzaPS3yQcPAaM6PE5zyzq1YcOGUhHZf6LtXkgCSrvdK/DYeQ8udt6DizfnPfpkDtyvrvQWkRDgc+A8nESxHviaqm730etlnczVjgOdnffgYuc9uPjyvPtVDUNVW0TkbuAdIBh4ylfJwhhjTM/0q4QBoKpvAm/6Ow5jjDFHG3Cd3r1sqb8D8BM778HFzntw8dl596s+DGOMMf3XYK9hGGOM8ZIlDGOMMV4ZlAkjECY4FJGnRKRYRLZ1KEsQkXdFZLf7N94tFxF51D3fLSIyt8NzbnL33y0iN3UoP01EtrrPeVSkfyxzJyKjROQDEdkhIttF5F63PKDPXUQiRGSdiHzmnveP3fIxIvKpG+vzIhLmloe7j3Pc7RkdjvUDt3yXiFzUobzffi5EJFhENonIG+7jgD9vEcl134ebRSTLLfPv+1xVB9UNZ7juHmAsEAZ8Bkz1d1wncR5nA3OBbR3KfgE86N5/EPi5e/8S4C1AgNOBT93yBGCv+zfevR/vblvn7ivucy/29zm7cQ0H5rr3Y3Cu25ka6OfuxhLt3g8FPnVjfAG4zi3/I/At9/63gT+6968DnnfvT3Xf8+HAGPezENzfPxfA94BngTfcxwF/3kAukHRMmV/f54OxhhEQExyq6kqg/JjiK4E/u/f/DFzVofxpdawF4kRkOHAR8K6qlqvqYeBdYLG7LVZV16rzznq6w7H8SlULVHWje78a2Ikz31hAn7sbf437MNS9KXAu8De3/Njzbvv3+BtwnvsL8krgOVVtVNV9QA7OZ6Lffi5EJA24FHjCfSwMgvM+Ab++zwdjwuh2gsMBLFVVC9z7hUCqe/9E59xVeV4n5f2K29wwB+fXdsCfu9sssxkoxvng7wEqVLXF3aVjrO3n526vBBLp+b9Hf/C/wP2Ax32cyOA4bwX+ISIbROROt8yv7/N+d+Ge6R2qqiISsGOmRSQaeBH4f6pa1bH5NVDPXVVbgdkiEge8DEz2c0g+JyKXAcWqukFEFvk7nj52lqoeEpEU4F0Rye640R/v88FYw+jRBIcDTJFb1cT9W+yWn+icuypP66S8XxCRUJxksVxVX3KLB8W5A6hqBfABcAZO00PbD7+Osbafn7t9KFBGz/89/G0BcIWI5OI0F50L/IbAP29U9ZD7txjnB8J8/P0+93fHTl/fcGpVe3E6vto6uab5O66TPJcMju70/iVHd4j9wr1/KUd3iK3TIx1i+3A6w+Ld+wnaeYfYJf4+XzcuwWlv/d9jygP63IFkIM69PwT4GLgM+CtHd/5+271/F0d3/r7g3p/G0Z2/e3E6fvv95wJYxJFO74A+byAKiOlwfzWw2N/vc7+/Cfz0n3EJzuiaPcBD/o7nJM9hBVAANOO0P96G01b7PrAbeK/DG0Nwlr7dA2wFMjsc51acDsAc4JYO5ZnANvc5v8OdFcDfN+AsnLbdLcBm93ZJoJ87MBPY5J73NuDf3PKx7gc/x/0SDXfLI9zHOe72sR2O9ZB7brvoMDKmv38uODphBPR5u+f3mXvb3haXv9/nNjWIMcYYrwzGPgxjjDEnwRKGMcYYr1jCMMYY4xVLGMYYY7xiCcMYY4xXLGEYA4hIqzsraNuty1lLReSbInJjL7xurogknepxjOkLNqzWGEBEalQ12g+vm4szZr60r1/bmJ6yGoYxXXBrAL9w1w1YJyLj3fKHReQ+9/494qzPsUVEnnPLEkTkFbdsrYjMdMsTReQf4qxp8QTOBVdtr/V19zU2i8hj7mSDwSKyTES2uTF81w//DMYAljCMaTPkmCapr3bYVqmqM3Cuhv3fTp77IDBHVWcC33TLfgxscst+iDOdCcC/A6tUdRrO/EDpACIyBfgqsEBVZwOtwPXAbGCkqk53Y/hTL56zMT1is9Ua46h3v6g7s6LD3193sn0LsFxEXgFeccvOAq4BUNV/ujWLWJyFr77klv9dRA67+58HnAasd2feHYIzsdzrwFgR+S3wd+AfJ3+Kxpwaq2EY0z09wf02l+LM4zMX5wv/ZH6ICfBnVZ3t3iap6sPqLHozC/gQp/byxEkc25heYQnDmO59tcPfNR03iEgQMEpVPwAewJlOOxpnNtnr3X0WAaWqWgWsBL7mll+MM4MoOBPKXeuufdDWBzLaHUEVpKovAv+Kk5SM8QtrkjLGMcRdza7N26raNrQ2XkS2AI3AkmOeFww8IyJDcWoJj6pqhYg8DDzlPq8OuMnd/8fAChHZjjNl9QEAVd0hIv+Ks8JaEM4sxHcB9cCf3DKAH/TeKRvTMzas1pgu2LBXY46wJiljjDFesRqGMcYYr1gNwxhjjFcsYRhjjPGKJQxjjDFesYRhjDHGK5YwjDHGeOX/A50W4KiwW4BbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label = 'avg')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label = 'min')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label = 'max')\n",
    "plt.legend(loc='upper left')\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Rewards')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Rendering Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "done = False\n",
    "state = env.reset()\n",
    "while not done:\n",
    "    action = np.argmax(q_table[get_discrete_state(state)])\n",
    "    next_state, _, done, _ = env.step(action)\n",
    "    state = next_state\n",
    "    env.render()\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python(spinningup)",
   "language": "python",
   "name": "spinningup"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
